1. Field of the Invention
This invention relates generally to the field of computer graphics and, more particularly, to high performance graphics systems.
2. Description of the Related Art
A computer system typically relies upon its graphics system for producing visual output on the computer screen or display device. Early graphics systems were only responsible for taking what the processor produced as output and displaying that output on the screen. In essence, they acted as simple translators or interfaces. Modem graphics systems, however, incorporate graphics processors with a great deal of processing power. They now act more like coprocessors rather than simple translators. This change is due to the recent increase in both the complexity and amount of data being sent to the display device. For example, modem computer displays have many more pixels, greater color depth, and are able to display images that are more complex with higher refresh rates than earlier models. Similarly, the images displayed are now more complex and may involve advanced techniques such as anti-aliasing and texture mapping.
As a result, without considerable processing power in the graphics system, the CPU would spend a great deal of time performing graphics calculations. This could rob the computer system of the processing power needed for performing other tasks associated with program execution and thereby dramatically reduce overall system performance. With a powerful graphics system, however, when the CPU is instructed to draw a box on the screen, the CPU is freed from having to compute the position and color of each pixel. Instead, the CPU may send a request to the video card stating: xe2x80x9cdraw a box at these coordinatesxe2x80x9d. The graphics system then draws the box, freeing the processor to perform other tasks.
Generally, a graphics system in a computer (also referred to as a graphics system) is a type of video adapter that contains its own processor to boost performance levels. These processors are specialized for computing graphical transformations, so they tend to achieve better results than the general-purpose CPU used by the computer system. In addition, they free up the computer""s CPU to execute other commands while the graphics system is handling graphics computations. The popularity of graphical applications, and especially multimedia applications, has made high performance graphics systems a common feature of computer systems. Most computer manufacturers now bundle a high performance graphics system with their systems.
Since graphics systems typically perform only a limited set of functions, they may be customized and therefore far more efficient at graphics operations than the computer""s general-purpose central processor. While early graphics systems were limited to performing two-dimensional (2D) graphics, their functionality has increased to support three-dimensional (3D) wire-frame graphics, 3D solids, and now includes support for three-dimensional (3D) graphics with textures and special effects such as advanced shading, fogging, alpha-blending, and specular highlighting.
The processing power of 3D graphics systems has been improving at a breakneck pace. A few years ago, shaded images of simple objects could only be rendered at a few frames per second, while today""s systems support rendering of complex objects at 60 Hz or higher. At this rate of increase, in the not too distant future, graphics systems will literally be able to render more pixels than a single human""s visual system can perceive. While this extra performance may be useable in multiple-viewer environments, it may be wasted in more common primarily single-viewer environments. Thus, a graphics system is desired which is capable of matching the variable nature of the human resolution system (i.e., capable of putting the quality where it is needed or most perceivable).
While the number of pixels is an important factor in determining graphics system performance, another factor of equal import is the quality of the image. For example, an image with a high pixel density may still appear unrealistic if edges within the image are too sharp or jagged (also referred to as xe2x80x9caliasedxe2x80x9d). One well-known technique to overcome these problems is anti-aliasing. Anti-aliasing involves smoothing the edges of objects by shading pixels along the borders of graphical elements. More specifically, anti-aliasing entails removing higher frequency components from an image before they cause disturbing visual artifacts. For example, anti-aliasing may soften or smooth high contrast edges in an image by forcing certain pixels to intermediate values (e.g., around the silhouette of a bright object superimposed against a dark background).
Another visual effect used to increase the realism of computer images is alpha blending. Alpha blending is a technique that controls the transparency of an object, allowing realistic rendering of translucent surfaces such as water or glass. Another effect used to improve realism is fogging. Fogging obscures an object as it moves away from the viewer. Simple fogging is a special case of alpha blending in which the degree of alpha changes with distance so that the object appears to vanish into a haze as the object moves away from the viewer. This simple fogging may also be referred to as xe2x80x9cdepth cueingxe2x80x9d or atmospheric attenuation, i.e., lowering the contrast of an object so that it appears less prominent as it recedes. Types of fogging that are more complex go beyond a simple linear function to provide relationships that are more complex between the level of translucence and an object""s distance from the viewer. Current state of the art software systems go even further by utilizing atmospheric models to provide low-lying fog with improved realism.
While the techniques listed above may dramatically improve the appearance of computer graphics images, they also have certain limitations. In particular, they may introduce their own aberrations and are typically limited by the density of pixels displayed on the display device.
As a result, a graphics system is desired which is capable of utilizing increased performance levels to increase not only the number of pixels rendered but also the quality of the image rendered. In addition, a graphics system is desired which is capable of utilizing increases in processing power to improve the results of graphics effects such as anti-aliasing.
Prior art graphics systems have generally fallen short of these goals. Prior art graphics systems use a conventional frame buffer for refreshing pixel/video data on the display. The frame buffer stores rows and columns of pixels that exactly correspond to respective row and column locations on the display. Prior art graphics system render 2D and/or 3D images or objects into the frame buffer in pixel form, and then read the pixels from the frame buffer during a screen refresh to refresh the display. Thus, the frame buffer stores the output pixels that are provided to the display. To reduce visual artifacts that may be created by refreshing the screen at the same time as the frame buffer is being updated, most graphics systems"" frame buffers are double-buffered.
To obtain images that are more realistic, some prior art graphics systems have gone further by generating more than one sample per pixel. As used herein, the term xe2x80x9csamplexe2x80x9d refers to calculated color information that indicates the color, depth (z), transparency, and potentially other information, of a particular point on an object or image. For example, a sample may comprise the following component values: a red value, a green value, a blue value, a z value, and an alpha value (e.g., representing the transparency of the sample). A sample may also comprise other information, e.g., a z-depth value, a blur value, an intensity value, brighter-than-bright information, and an indicator that the sample consists partially or completely of control information rather than color information (i.e., xe2x80x9csample control informationxe2x80x9d). By calculating more samples than pixels (i.e., super-sampling), a more detailed image is calculated than can be displayed on the display device. For example, a graphics system may calculate four samples for each pixel to be output to the display device. After the samples are calculated, they are then combined or filtered to form the pixels that are stored in the frame buffer and then conveyed to the display device. Using pixels formed in this manner may create a more realistic final image because overly abrupt changes in the image may be smoothed by the filtering process.
These prior art super-sampling systems typically generate a number of samples that are far greater than the number of pixel locations on the display. These prior art systems typically have rendering processors that calculate the samples and store them into a render buffer. Filtering hardware then reads the samples from the render buffer, filters the samples to create pixels, and then stores the pixels in a traditional frame buffer. The traditional frame buffer is typically double-buffered, with one side being used for refreshing the display device while the other side is updated by the filtering hardware. Once the samples have been filtered, the resulting pixels are stored in a traditional frame buffer that is used to refresh the display device. These systems, however, have generally suffered from limitations imposed by the conventional frame buffer and by the added latency caused by the render buffer and filtering. Therefore, an improved graphics system is desired which includes the benefits of pixel super-sampling while avoiding the drawbacks of the conventional frame buffer.
Furthermore, previous super-sampled graphics systems attempt to generate every output pixel value by filtering the sample values, i.e., a computationally intensive process. In most cases, these traditional graphics systems are able to cope with typical display resolutions and/or refresh rates. However, in some instances, the graphics system may be requested to generate pixels corresponding to a display resolution and/or a refresh rate that is beyond the computational powers of the graphics system. Thus, a graphics system is desired that is capable of generating less than the requested number of pixels using filtering of sample values. The remaining of the requested pixels may then be generated by some other means.
In addition, a graphics system is desired that is capable of generating the requested pixels by filtering samples for one area of the display screen where a picture of high quality is desired and generating less than the requested number of pixels by filtering samples for other areas of the display screen where a lower quality picture is sufficient.
The present invention contemplates a computer graphics system that utilizes a sample buffer and a sample-to-pixel calculation unit for refreshing the display. In one embodiment, the graphics system may have a graphics processor, a super-sampled sample buffer, and a sample-to-pixel calculation unit. In another embodiment, the graphics system may have a graphics processor, a super-sampled sample buffer, a filtering unit, and an interpolation unit.
The graphics processor generates a plurality of samples and stores them into a sample buffer. The graphics processor preferably generates and stores more than one sample for at least a subset of the pixel locations on the display. Thus, the sample buffer may be a super-sampled sample buffer which stores a number of samples that, in some embodiments, may be far greater than the number of pixel locations on the display. In other embodiments, the total number of samples may be closer to, equal to, or even less than the total number of pixel locations on the display device, but the samples may be more densely positioned in certain areas and less densely positioned in other areas.
The sample-to-pixel calculation unit is configured to read the samples from the super-sampled sample buffer and filter or convolve the samples into respective output pixels, wherein the output pixels are then provided to refresh the display. Note that, as used herein, the terms xe2x80x9cfilterxe2x80x9d and xe2x80x9cconvolvexe2x80x9d are used interchangeably and refer to mathematically manipulating one or more samples to generate a pixel (e.g., by averaging, by applying a convolution function, by summing, by applying a filtering function, by weighting the samples and then manipulating them, by applying a randomized function, etc.). The sample-to-pixel calculation unit selects one or more samples and filters them to generate an output pixel. Note that the number of samples selected and or filtered by the sample-to-pixel calculation unit may be one or, in the preferred embodiment, greater than one.
The graphics system may operate without a conventional frame buffer, i.e., the graphics system does not utilize a conventional frame buffer that stores the actual pixel values that are being refreshed on the display. Note that some displays may have internal frame buffers, but these are considered an integral part of the display device, not the graphics system. Thus, the sample-to-pixel calculation units may calculate each pixel for each screen refresh on a real time basis. As used herein, the term xe2x80x9creal-timexe2x80x9d refers to a function that is performed at or near the display device""s refresh rate.xe2x80x9d xe2x80x9cOn-the-flyxe2x80x9d means at, near, or above the human visual system""s perception capabilities for motion fusion (how often a picture must be changed to give the illusion of continuous motion) and flicker fusion (how often light intensity must be changed to give the illusion of continuous). These concepts are further described in the book xe2x80x9cSpatial Visionxe2x80x9d by Russel L. De Valois and Karen K. De Valois, Oxford University Press, 1988.
The sample-to-pixel calculation unit may be programmed to generate a first subset of the output pixels (filtered pixels) by filtering the rendered samples and to generate a second subset of the output pixels (interpolated pixels) by interpolating the filtered pixels and/or the rendered samples. A different ratio of filtered to interpolated pixels may be generated for different regions of the display or the same ratio may be used for the entire display. In one embodiment, the ratio may vary according to the location of the output pixel, for example, the distance of the output pixel from a viewer""s point of foveation. As used herein, the term xe2x80x9cpoint of foveationxe2x80x9d refers to a point (e.g., on a display screen) on which the center of a viewer""s eyes"" field of vision is focused. This point may move as the viewer""s eyes move. For example, the point of foveation (which moves as the viewer""s eyes move) may be located at the exact center of the display screen when the viewer is focusing on a small object displayed at the center of the screen. Note that in the cases where a different image is displayed for each eye (e.g., for a 3D effect), two points of foveation may exist, one for each eye. In those cases, the ratio of filtered to interpolated pixels may vary according to the location of both points of foveation.
In one embodiment, the first and second subsets of pixels may comprise every other pixel in a screen-space line of the display. In other embodiments, the first subset of pixels may comprise one out of every three pixels, one out of every four pixels, etc, or any other ratio of first to second subset of pixels may be used.
In one embodiment, each interpolated pixel may be generated by interpolating two closest filtered pixels to the pixel. The two closest filtered pixels may or may not be along the same scan line of the display as the pixel. In an embodiment where the pixels are along the same scan line, only one pixel from the first subset needs to be cached at any time, and the sample-to-pixel calculation unit may alternately generate a filtered pixel and an interpolated pixel.
In other embodiments, the sample-to-pixel calculation unit may generate each interpolated pixel by interpolating: a closest filtered pixel and a closest sample to the interpolated pixel; two closest filtered pixels and two closest samples to the interpolated pixel; or two closest filtered pixels to the interpolated pixel. Note that in all the embodiments, the filtered pixels may or may not be along the same scan line as the interpolated pixel, and that the interpolation may be performed along a scan line (1D interpolation) or in the plane (2D interpolation).
The human visual system has varying levels of acuity, with the highest level of acuity occurring near the foveal pit of the retina. The foveal region receives light from the point of foveation and typically accounts for only a few degrees at the center of a human""s of field of vision. Thus, to best match the human visual system, the graphics system may be, in some embodiments, configured to detect where the viewer""s point of foveation is relative to the display device. This allows the graphics system to match the ratio of first to second subset of pixels to the human eye""s acuity. Thus, a higher number of filtered/lower number of interpolated pixels (and more processing power) may be allocated to areas of the display device that will be perceived by the highest acuity regions of the human visual system. Similarly, a lower number of filtered/higher number of interpolated pixels and processing power will be devoted to regions that will be perceived by the lower acuity regions of the human visual system. Note, however, it is not just the density of rods and cones in the eye that may be matched. Other factors may also influence the perception of the human visual system, including the lens system, chromatic aberrations, and the neural pathways to the eye. For the purposes of matching computer displays to human retinal perception, the human brain""s processing limits for visual input provides a useful target that future graphics systems may strive to match or exceed.
This type of graphics system may be implemented in a number of different ways. For example, eye-tracking sensors may be used to determine in what direction the viewer""s eyes are directed. This may provide data with which to predict where the viewer""s point of foveation is. Typically, head-mounted eye-tracking sensors may use an additional eye-tracking sensor. Taken together, the eye- and head-tracking sensors can provide useful information about the position and movement of a viewer""s point of foveation relative to the display device. Even further accuracy may be obtained using two eye-tracking sensors (i.e., one for each of the viewer""s eyes). Thus, two points of foveation may be detected for each viewer. Furthermore, in some configurations multiple viewers may each have their points of foveation detected. Other configurations may utilize a hand-tracking sensor (e.g., pointing wand or data glove) in combination with head- and or eye-tracking sensors. Another configuration may utilize a head-mounted display with various motion, direction, eye tracking, and or head-tracking sensors. A higher number of filtered/lower number of interpolated pixels (and more processing power) may be allocated to a region of a predetermined size centered at the calculated point of foveation to compensate for inaccuracies in the sensors. Note that, as used herein, the term xe2x80x9cgaze tracking unitxe2x80x9d refers to any combination of eye-tracking, head-tracking, hand tracking, and or body tracking sensors that provide information concerning one or more viewers"" points of foveation (there can be two points of foveation for each viewer). Examples of gaze tracking units may include one or more of the following: video cameras, xe2x80x9cEMGxe2x80x9d sensors that detect electrical currents in muscles, an eye-and-head tracker, an eye tracker, a head tracker, a hand tracker, a data glove, a wand, a data suit, a mouse, a body position sensor, a body position sensing chair, motion sensors, pressure sensors, acoustic sensors, and infra-red scanners/sensors. In other embodiments, the system may assume that the viewer""s point of foveation is located at a fixed location near the center of the screen, or at a varying point of interest on the display created by the software application being executed.
The graphics processor may be similarly configured to vary the number of filtered and number of interpolated pixels (and the processing power) for different regions of the displayed image. These different ratios may be positioned based on the point of interest, cursor position, eye tracking, head tracking, etc. In other embodiments, the ratio of the filtered to the interpolated pixels may be varied on a scan line basis, a per-pixel basis, or a per-frame region basis.